{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## **Corrective RAG**\n",
        "\n",
        "Corrective Retrieval-Augmented Generation (Corrective RAG or CRAG) is a method that improves the accuracy of generated responses by evaluating and correcting retrieved documents before using them. It works as follows:\n",
        "\n",
        "**Correct:** If documents are relevant, they are refined to remove unnecessary parts and used for generation.\n",
        "\n",
        "**Incorrect:** If documents are irrelevant, they are discarded, and the system retrieves additional information, like using web search.\n",
        "(**Note:** In our case, we are not performing this step.)\n",
        "\n",
        "**Ambiguous:** If it's unclear, the system combines retrieved and web-searched information for a balanced response.\n",
        "\n",
        "Research Paper: [Corrective RAG](https://arxiv.org/pdf/2401.15884)"
      ],
      "metadata": {
        "id": "eRpYIXC-LW0u"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Initial Setup**"
      ],
      "metadata": {
        "id": "YWLRt2mi0rXU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! pip install --q athina chromadb langgraph"
      ],
      "metadata": {
        "id": "g6-0ne6BAZWW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dff6c06e-a3c3-4896-a851-ed56a231e435"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/83.5 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m83.5/83.5 kB\u001b[0m \u001b[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mqVkXJwJA945"
      },
      "outputs": [],
      "source": [
        "# ! pip install --q athina datasets langchain_community tiktoken langchain-openai langchainhub chromadb langchain langgraph tavily-python"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "from google.colab import userdata\n",
        "os.environ[\"OPENAI_API_KEY\"] = userdata.get('OPENAI_API_KEY')\n",
        "os.environ[\"TAVILY_API_KEY\"] = userdata.get('TAVILY_API_KEY')\n",
        "os.environ['ATHINA_API_KEY'] = userdata.get('ATHINA_API_KEY')"
      ],
      "metadata": {
        "id": "LcIN7gNdB16k"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Indexing**"
      ],
      "metadata": {
        "id": "QiWNs9Nz0quz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# load embedding model\n",
        "from langchain_openai import OpenAIEmbeddings\n",
        "embeddings = OpenAIEmbeddings()"
      ],
      "metadata": {
        "id": "4e8jZIs50pjA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# load data\n",
        "from langchain.document_loaders import CSVLoader\n",
        "loader = CSVLoader(\"./context.csv\")\n",
        "documents = loader.load()"
      ],
      "metadata": {
        "id": "TKiUzb3607j6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# split documents\n",
        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
        "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)\n",
        "documents = text_splitter.split_documents(documents)"
      ],
      "metadata": {
        "id": "DAxUi8unnJq-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create vectorstore\n",
        "from langchain.vectorstores import Chroma\n",
        "vectorstore = Chroma.from_documents(documents, embeddings)"
      ],
      "metadata": {
        "id": "AiOJeXQYCWp0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Retriever**"
      ],
      "metadata": {
        "id": "gx62vHdsMPH-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create retriever\n",
        "retriever = vectorstore.as_retriever()"
      ],
      "metadata": {
        "id": "q18W19uU4OKH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Document Grader**\n",
        "The document grader evaluates whether a document is relevant to the given query."
      ],
      "metadata": {
        "id": "F6POsWW5MSuo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create grader for doc retriever\n",
        "from langchain_core.prompts import ChatPromptTemplate\n",
        "from langchain_core.pydantic_v1 import BaseModel, Field\n",
        "from langchain_openai import ChatOpenAI\n",
        "\n",
        "# defining a data class for the grader\n",
        "class GradeDocuments(BaseModel):\n",
        "    binary_score: str = Field(\n",
        "        description=\"Documents are relevant to the question, 'yes' or 'no'\"\n",
        "    )\n",
        "\n",
        "# LLM with function call\n",
        "llm = ChatOpenAI(temperature=0)\n",
        "structured_llm_grader = llm.with_structured_output(GradeDocuments)\n",
        "\n",
        "# Prompt for the grader\n",
        "system = \"\"\"You are a grader assessing relevance of a retrieved document to a user question. \\n\n",
        "    It does not need to be a stringent test. The goal is to filter out erroneous retrievals. \\n\n",
        "    If the document contains keyword(s) or semantic meaning related to the user question, grade it as relevant. \\n\n",
        "    Give a binary score 'yes' or 'no' score to indicate whether the document is relevant to the question.\"\"\"\n",
        "grade_prompt = ChatPromptTemplate.from_messages(\n",
        "    [\n",
        "        (\"system\", system),\n",
        "        (\"human\", \"Retrieved document: \\n\\n {document} \\n\\n User question: {question}\"),\n",
        "    ]\n",
        ")\n",
        "\n",
        "retrieval_grader = grade_prompt | structured_llm_grader"
      ],
      "metadata": {
        "id": "ZZt6fLGf4SLG",
        "outputId": "291428ea-f8cd-4bc5-8929-016739a9ff22",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/langchain_core/_api/beta_decorator.py:87: LangChainBetaWarning: The method `ChatOpenAI.with_structured_output` is in beta. It is actively being worked on, so the API may change.\n",
            "  warn_beta(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# run grader\n",
        "question = \"how does interlibrary loan work\"\n",
        "docs = retriever.get_relevant_documents(question)\n",
        "print(retrieval_grader.invoke({\"question\": question, \"document\": docs}))"
      ],
      "metadata": {
        "id": "qOvrTWya4Wka",
        "outputId": "2fbade7a-4ad1-4815-a530-5d275507f250",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:119: LangChainDeprecationWarning: The method `BaseRetriever.get_relevant_documents` was deprecated in langchain-core 0.1.46 and will be removed in 0.3.0. Use invoke instead.\n",
            "  warn_deprecated(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "binary_score='yes'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **RAG Chain**"
      ],
      "metadata": {
        "id": "bNkpwjkGMZhS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create document chain\n",
        "from langchain import hub\n",
        "from langchain_core.output_parsers import StrOutputParser\n",
        "from langchain.prompts import ChatPromptTemplate\n",
        "\n",
        "template = \"\"\"\"\n",
        "You are a helpful assistant that answers questions based on the following context.'\n",
        "Use the provided context to answer the question.\n",
        "Context: {context}\n",
        "Question: {question}\n",
        "Answer:\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "prompt = ChatPromptTemplate.from_template(template)\n",
        "llm = ChatOpenAI(temperature=0)\n",
        "\n",
        "def format_docs(docs):\n",
        "    return \"\\n\\n\".join(doc.page_content for doc in docs)\n",
        "\n",
        "\n",
        "rag_chain = prompt | llm | StrOutputParser()"
      ],
      "metadata": {
        "id": "3Mx1FkPm4gtD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# response\n",
        "generation = rag_chain.invoke({\"context\": docs, \"question\": question})\n",
        "generation"
      ],
      "metadata": {
        "id": "eJ5VsFWlpDSk",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "e22f1099-7286-4077-deed-e87a6a353db8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Interlibrary loan (ILL) is a service that allows patrons of one library to borrow physical materials or receive electronic documents from another library. After receiving a request from a patron, the borrowing library identifies potential lending libraries with the desired item. The lending library then delivers the item physically or electronically to the borrowing library, who then delivers it to their patron. If necessary, arrangements are made for the return of the item. Fees may accompany interlibrary loan services, and the majority of requests are now managed through semi-automated systems. Libraries negotiate for ILL eligibility, and with the increasing demand for digital materials, they are exploring the legal, technical, and licensing aspects of lending and borrowing ebooks through interlibrary loan.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Web Search**"
      ],
      "metadata": {
        "id": "FIRfqwSUMbYi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# define web search\n",
        "from langchain_community.tools.tavily_search import TavilySearchResults\n",
        "web_search_tool = TavilySearchResults(k=3)"
      ],
      "metadata": {
        "id": "hFEa3BB2CZeh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# # sample web search\n",
        "# web_search_tool.invoke('USA election 2024')"
      ],
      "metadata": {
        "id": "bAzeatF-pZUH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Create Graph**"
      ],
      "metadata": {
        "id": "aqidnV4PpdVU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Define Graph State**"
      ],
      "metadata": {
        "id": "6y2J-am8phxH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# define a data class for state\n",
        "from typing import List\n",
        "from typing_extensions import TypedDict\n",
        "\n",
        "class GraphState(TypedDict):\n",
        "    question: str\n",
        "    generation: str\n",
        "    web_search: str\n",
        "    documents: List[str]"
      ],
      "metadata": {
        "id": "hk3cW_wjCcVU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Build Graph**"
      ],
      "metadata": {
        "id": "RLTPwOjLqLcv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# define graph steps\n",
        "from langchain.schema import Document\n",
        "\n",
        "# node function for retrieval\n",
        "def retrieve(state):\n",
        "\n",
        "    print(\"---RETRIEVE---\")\n",
        "    question = state[\"question\"]\n",
        "\n",
        "    # Retrieval\n",
        "    documents = retriever.invoke(question)\n",
        "    return {\"documents\": documents, \"question\": question}\n",
        "\n",
        "# node function for generation\n",
        "def generate(state):\n",
        "\n",
        "    print(\"---GENERATE---\")\n",
        "    question = state[\"question\"]\n",
        "    documents = state[\"documents\"]\n",
        "\n",
        "    # RAG generation\n",
        "    generation = rag_chain.invoke({\"context\": documents, \"question\": question})\n",
        "    return {\"documents\": documents, \"question\": question, \"generation\": generation}\n",
        "\n",
        "# node function for check_relevance\n",
        "def grade_documents(state):\n",
        "\n",
        "    print(\"---CHECK DOCUMENT RELEVANCE TO QUESTION---\")\n",
        "    question = state[\"question\"]\n",
        "    documents = state[\"documents\"]\n",
        "\n",
        "    # Score each doc\n",
        "    filtered_docs = []\n",
        "    web_search = \"No\"\n",
        "    for d in documents:\n",
        "        score = retrieval_grader.invoke(\n",
        "            {\"question\": question, \"document\": d.page_content}\n",
        "        )\n",
        "        grade = score.binary_score\n",
        "        if grade == \"yes\":\n",
        "            print(\"---GRADE: DOCUMENT RELEVANT---\")\n",
        "            filtered_docs.append(d)\n",
        "        else:\n",
        "            print(\"---GRADE: DOCUMENT NOT RELEVANT---\")\n",
        "            web_search = \"Yes\"\n",
        "            continue\n",
        "    return {\"documents\": filtered_docs, \"question\": question, \"web_search\": web_search}\n",
        "\n",
        "\n",
        "# node function for web search\n",
        "def web_search(state):\n",
        "\n",
        "    print(\"---WEB SEARCH---\")\n",
        "    question = state[\"question\"]\n",
        "    documents = state[\"documents\"]\n",
        "\n",
        "    # Web search\n",
        "    docs = web_search_tool.invoke({\"query\": question})\n",
        "    web_results = \"\\n\".join([d[\"content\"] for d in docs])\n",
        "    web_results = Document(page_content=web_results)\n",
        "    documents.append(web_results)\n",
        "\n",
        "    return {\"documents\": documents, \"question\": question}\n",
        "\n",
        "# node function for decision\n",
        "def decide_to_generate(state):\n",
        "\n",
        "    print(\"---ASSESS GRADED DOCUMENTS---\")\n",
        "    state[\"question\"]\n",
        "    web_search = state[\"web_search\"]\n",
        "    state[\"documents\"]\n",
        "\n",
        "    if web_search == \"Yes\":\n",
        "        print(\"---DECISION: WEB SEARCH---\")\n",
        "        return \"web_search\"\n",
        "    else:\n",
        "        print(\"---DECISION: GENERATE---\")\n",
        "        return \"generate\""
      ],
      "metadata": {
        "id": "X17xcid8Cg97"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Build graph\n",
        "from langgraph.graph import END, StateGraph, START\n",
        "\n",
        "# Graph\n",
        "workflow = StateGraph(GraphState)\n",
        "\n",
        "# Define the nodes\n",
        "workflow.add_node(\"retrieve\", retrieve)  # retrieve\n",
        "workflow.add_node(\"grade_documents\", grade_documents)  # grade documents\n",
        "workflow.add_node(\"generate\", generate)  # generatae\n",
        "workflow.add_node(\"web_search_node\", web_search)  # web search\n",
        "\n",
        "# Build graph\n",
        "workflow.add_edge(START, \"retrieve\")\n",
        "workflow.add_edge(\"retrieve\", \"grade_documents\")\n",
        "workflow.add_conditional_edges(\n",
        "    \"grade_documents\",\n",
        "    decide_to_generate,\n",
        "    {\n",
        "        \"web_search\": \"web_search_node\",\n",
        "        \"generate\": \"generate\",\n",
        "    },\n",
        ")\n",
        "\n",
        "workflow.add_edge(\"web_search_node\", \"generate\")\n",
        "workflow.add_edge(\"generate\", END)\n",
        "\n",
        "# Compile\n",
        "app = workflow.compile()"
      ],
      "metadata": {
        "id": "UBfHWkUuCma-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# example 1 where documents are relevant\n",
        "from pprint import pprint\n",
        "\n",
        "inputs = {\"question\": \"how does interlibrary loan work\"}\n",
        "for output in app.stream(inputs):\n",
        "    for key, value in output.items():\n",
        "        pprint(f\"Node '{key}':\")\n",
        "    pprint(\"\\n---\\n\")\n",
        "\n",
        "\n",
        "pprint(value[\"generation\"])"
      ],
      "metadata": {
        "id": "enXoiLgeCo8a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "68791866-fa7a-4946-d4d8-effa686350f3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---RETRIEVE---\n",
            "\"Node 'retrieve':\"\n",
            "'\\n---\\n'\n",
            "---CHECK DOCUMENT RELEVANCE TO QUESTION---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---ASSESS GRADED DOCUMENTS---\n",
            "---DECISION: GENERATE---\n",
            "\"Node 'grade_documents':\"\n",
            "'\\n---\\n'\n",
            "---GENERATE---\n",
            "\"Node 'generate':\"\n",
            "'\\n---\\n'\n",
            "('Interlibrary loan works by allowing patrons of one library to borrow '\n",
            " 'physical materials or receive electronic documents from another library that '\n",
            " 'holds the desired item. The borrowing library identifies potential lending '\n",
            " 'libraries, which then deliver the item either physically or electronically. '\n",
            " 'The borrowing library receives the item, delivers it to their patron, and '\n",
            " 'arranges for its return if necessary. Fees may accompany interlibrary loan '\n",
            " 'services, and the majority of requests are now managed through '\n",
            " 'semi-automated systems.')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# example 2 where documents are not relevant\n",
        "from pprint import pprint\n",
        "\n",
        "inputs = {\"question\": \"What is Retrieval-Augmented Generation (RAG)?\"}\n",
        "for output in app.stream(inputs):\n",
        "    for key, value in output.items():\n",
        "        pprint(f\"Node '{key}':\")\n",
        "    pprint(\"\\n---\\n\")\n",
        "\n",
        "pprint(value[\"generation\"])"
      ],
      "metadata": {
        "id": "UQk0nEZvrNoJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4aafa5f3-4f49-46c9-975c-5980b1c878f2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---RETRIEVE---\n",
            "\"Node 'retrieve':\"\n",
            "'\\n---\\n'\n",
            "---CHECK DOCUMENT RELEVANCE TO QUESTION---\n",
            "---GRADE: DOCUMENT NOT RELEVANT---\n",
            "---GRADE: DOCUMENT NOT RELEVANT---\n",
            "---GRADE: DOCUMENT NOT RELEVANT---\n",
            "---GRADE: DOCUMENT NOT RELEVANT---\n",
            "---ASSESS GRADED DOCUMENTS---\n",
            "---DECISION: WEB SEARCH---\n",
            "\"Node 'grade_documents':\"\n",
            "'\\n---\\n'\n",
            "---WEB SEARCH---\n",
            "\"Node 'web_search_node':\"\n",
            "'\\n---\\n'\n",
            "---GENERATE---\n",
            "\"Node 'generate':\"\n",
            "'\\n---\\n'\n",
            "('Retrieval-Augmented Generation (RAG) is a technique that enhances generative '\n",
            " 'artificial intelligence models by allowing them to access and reference '\n",
            " 'external knowledge bases, such as specific documents or databases, before '\n",
            " 'generating a response. This process helps improve the relevance and accuracy '\n",
            " \"of the generated text by incorporating information beyond the model's own \"\n",
            " 'training data.')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Preparing Data for Evaluation**"
      ],
      "metadata": {
        "id": "1GWUjEskYM6F"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a dataframe to store the question, context, and response\n",
        "inputs = {\"question\": \"what are points on a mortgage?\"}\n",
        "outputs = []\n",
        "expected_response = \"Points, sometimes also called a 'discount point', are a form of pre-paid interest .\"\n",
        "\n",
        "for output in app.stream(inputs):\n",
        "    for key, value in output.items():\n",
        "        if key == \"generate\":\n",
        "            question = value[\"question\"]\n",
        "            documents = value[\"documents\"]\n",
        "            generation = value[\"generation\"]\n",
        "\n",
        "            context = \"\\n\".join(doc.page_content for doc in documents)\n",
        "\n",
        "            # Append the result\n",
        "            outputs.append({\n",
        "                \"query\": question,\n",
        "                \"context\": context,\n",
        "                \"response\": generation,\n",
        "                \"expected_response\": expected_response\n",
        "            })\n"
      ],
      "metadata": {
        "id": "0lvr2-cfC3yo",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e4f6c2f7-9c32-4d85-85ac-6ad8180c874b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---RETRIEVE---\n",
            "---CHECK DOCUMENT RELEVANCE TO QUESTION---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---GRADE: DOCUMENT RELEVANT---\n",
            "---ASSESS GRADED DOCUMENTS---\n",
            "---DECISION: GENERATE---\n",
            "---GENERATE---\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Convert to DataFrame\n",
        "import pandas as pd\n",
        "df = pd.DataFrame(outputs)"
      ],
      "metadata": {
        "id": "n9OTspwkr6BJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df"
      ],
      "metadata": {
        "id": "yREUX0YDsDj9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 202
        },
        "outputId": "5e76555b-43fb-4375-f050-92dfba1ef949"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                            query  \\\n",
              "0  what are points on a mortgage?   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               context  \\\n",
              "0  context: [\"Discount points, also called mortgage points or simply points, are a form of pre-paid interest available in the United States when arranging a mortgage. One point equals one percent of the loan amount. By charging a borrower points, a lender effectively increases the yield on the loan above the amount of the stated interest rate.  Borrowers can offer to pay a lender points as a method to reduce the interest rate on the loan, thus obtaining a lower monthly payment in exchange for t...   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              response  \\\n",
              "0  Points on a mortgage, also known as discount points, are a form of pre-paid interest that borrowers can pay to lenders in order to reduce the interest rate on the loan. One point equals one percent of the loan amount. By paying points, borrowers can obtain a lower monthly payment in exchange for this upfront payment. Points can also be used to reduce the monthly payment for the purpose of qualifying for a loan based on monthly income. It is important to note that points are different from or...   \n",
              "\n",
              "                                                                     expected_response  \n",
              "0  Points, sometimes also called a 'discount point', are a form of pre-paid interest .  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b025588d-080d-4b5a-87cf-33d7919242a2\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>query</th>\n",
              "      <th>context</th>\n",
              "      <th>response</th>\n",
              "      <th>expected_response</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>what are points on a mortgage?</td>\n",
              "      <td>context: [\"Discount points, also called mortgage points or simply points, are a form of pre-paid interest available in the United States when arranging a mortgage. One point equals one percent of the loan amount. By charging a borrower points, a lender effectively increases the yield on the loan above the amount of the stated interest rate.  Borrowers can offer to pay a lender points as a method to reduce the interest rate on the loan, thus obtaining a lower monthly payment in exchange for t...</td>\n",
              "      <td>Points on a mortgage, also known as discount points, are a form of pre-paid interest that borrowers can pay to lenders in order to reduce the interest rate on the loan. One point equals one percent of the loan amount. By paying points, borrowers can obtain a lower monthly payment in exchange for this upfront payment. Points can also be used to reduce the monthly payment for the purpose of qualifying for a loan based on monthly income. It is important to note that points are different from or...</td>\n",
              "      <td>Points, sometimes also called a 'discount point', are a form of pre-paid interest .</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b025588d-080d-4b5a-87cf-33d7919242a2')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-b025588d-080d-4b5a-87cf-33d7919242a2 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-b025588d-080d-4b5a-87cf-33d7919242a2');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "  <div id=\"id_a569c22e-47fe-4e61-8e8e-c92f1ddd0faa\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_a569c22e-47fe-4e61-8e8e-c92f1ddd0faa button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 1,\n  \"fields\": [\n    {\n      \"column\": \"query\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"what are points on a mortgage?\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"context\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"context: [\\\"Discount points, also called mortgage points or simply points, are a form of pre-paid interest available in the United States when arranging a mortgage. One point equals one percent of the loan amount. By charging a borrower points, a lender effectively increases the yield on the loan above the amount of the stated interest rate.  Borrowers can offer to pay a lender points as a method to reduce the interest rate on the loan, thus obtaining a lower monthly payment in exchange for this\\npoints is the concept of the 'no closing cost loan', in which the consumer accepts a higher interest rate in return for the lender paying the loan's closing costs up front.  In some cases a purchaser can negotiate with the seller to get them to pay seller's points which can be used to pay mortgage points.\\\\n\\\\n\\\\n== References ==\\\\n\\\\n\\\\n== External links ==\\\\nirs.gov/publications/p936 \\u2013 IRS Form 936 defines a point for the purpose of deducting mortgage interest for U.S. income taxes\\\"\\nup-front payment. For each point purchased, the loan rate is typically reduced by anywhere from 1/8% (0.125%) to 1/4% (0.25%).Selling the property or refinancing prior to this break-even point will result in a net financial loss for the buyer while keeping the loan for longer than this break-even point will result in a net financial savings for the buyer. Accordingly, if the intention is to buy and sell the property or refinance, paying points will cost more than just paying the higher interest\\nrate.Points may also be purchased to reduce the monthly payment for the purpose of qualifying for a loan.  Loan qualification based on monthly income versus the monthly loan payment may sometimes only be achievable by reducing the monthly payment through the purchasing of points to buy down the interest rate, thereby reducing the monthly loan payment.\\\\nDiscount points may be different from origination fee, mortgage arrangement fee or broker fee. Discount points are always used to buy down the\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"response\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"Points on a mortgage, also known as discount points, are a form of pre-paid interest that borrowers can pay to lenders in order to reduce the interest rate on the loan. One point equals one percent of the loan amount. By paying points, borrowers can obtain a lower monthly payment in exchange for this upfront payment. Points can also be used to reduce the monthly payment for the purpose of qualifying for a loan based on monthly income. It is important to note that points are different from origination fees, mortgage arrangement fees, or broker fees.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"expected_response\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"Points, sometimes also called a 'discount point', are a form of pre-paid interest .\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Convert to dictionary\n",
        "df_dict = df.to_dict(orient='records')\n",
        "\n",
        "# Convert context to list\n",
        "for record in df_dict:\n",
        "    if not isinstance(record.get('context'), list):\n",
        "        if record.get('context') is None:\n",
        "            record['context'] = []\n",
        "        else:\n",
        "            record['context'] = [record['context']]"
      ],
      "metadata": {
        "id": "lnTZAxQlNfYu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Evaluation in Athina AI**\n",
        "\n",
        "We will use **Context Precision** eval here. It Evaluates whether all of the ground-truth relevant items present in the contexts are ranked higher or not. Please refer to our [documentation](https://docs.athina.ai/api-reference/evals/preset-evals/overview) for further details."
      ],
      "metadata": {
        "id": "X_2BgxnBNjWy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# set api keys for Athina evals\n",
        "from athina.keys import AthinaApiKey, OpenAiApiKey\n",
        "OpenAiApiKey.set_key(os.getenv('OPENAI_API_KEY'))\n",
        "AthinaApiKey.set_key(os.getenv('ATHINA_API_KEY'))"
      ],
      "metadata": {
        "id": "gd4vFX0GNmiT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# load dataset\n",
        "from athina.loaders import Loader\n",
        "dataset = Loader().load_dict(df_dict)"
      ],
      "metadata": {
        "id": "DC-EGzgj3Odm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# evaluate\n",
        "from athina.evals import RagasContextPrecision\n",
        "RagasContextPrecision(model=\"gpt-4o\").run_batch(data=dataset).to_df()"
      ],
      "metadata": {
        "id": "LIygZMZ53MNc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 497
        },
        "outputId": "87e653b1-7176-4bac-a371-50904b82e506"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "evaluating with [context_precision]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 1/1 [00:01<00:00,  1.28s/it]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "You can view your dataset at: https://app.athina.ai/develop/2c0c4475-a30e-4d2c-92f9-c57f494ba947\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                            query  \\\n",
              "0  what are points on a mortgage?   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               context  \\\n",
              "0  [context: [\"Discount points, also called mortgage points or simply points, are a form of pre-paid interest available in the United States when arranging a mortgage. One point equals one percent of the loan amount. By charging a borrower points, a lender effectively increases the yield on the loan above the amount of the stated interest rate.  Borrowers can offer to pay a lender points as a method to reduce the interest rate on the loan, thus obtaining a lower monthly payment in exchange for ...   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              response  \\\n",
              "0  Points on a mortgage, also known as discount points, are a form of pre-paid interest that borrowers can pay to lenders in order to reduce the interest rate on the loan. One point equals one percent of the loan amount. By paying points, borrowers can obtain a lower monthly payment in exchange for this upfront payment. Points can also be used to reduce the monthly payment for the purpose of qualifying for a loan based on monthly income. It is important to note that points are different from or...   \n",
              "\n",
              "                                                                     expected_response  \\\n",
              "0  Points, sometimes also called a 'discount point', are a form of pre-paid interest .   \n",
              "\n",
              "              display_name failed  \\\n",
              "0  Ragas Context Precision   None   \n",
              "\n",
              "                                                                                                                                                                         grade_reason  \\\n",
              "0  This metric evaluates whether all of the ground-truth relevant items present in the context are ranked higher or not. Ideally all the relevant chunks must appear at the top ranks   \n",
              "\n",
              "   runtime   model  ragas_context_precision  \n",
              "0     1674  gpt-4o                      1.0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-2e9bc878-68de-4dde-a0c3-575d75b95e7b\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>query</th>\n",
              "      <th>context</th>\n",
              "      <th>response</th>\n",
              "      <th>expected_response</th>\n",
              "      <th>display_name</th>\n",
              "      <th>failed</th>\n",
              "      <th>grade_reason</th>\n",
              "      <th>runtime</th>\n",
              "      <th>model</th>\n",
              "      <th>ragas_context_precision</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>what are points on a mortgage?</td>\n",
              "      <td>[context: [\"Discount points, also called mortgage points or simply points, are a form of pre-paid interest available in the United States when arranging a mortgage. One point equals one percent of the loan amount. By charging a borrower points, a lender effectively increases the yield on the loan above the amount of the stated interest rate.  Borrowers can offer to pay a lender points as a method to reduce the interest rate on the loan, thus obtaining a lower monthly payment in exchange for ...</td>\n",
              "      <td>Points on a mortgage, also known as discount points, are a form of pre-paid interest that borrowers can pay to lenders in order to reduce the interest rate on the loan. One point equals one percent of the loan amount. By paying points, borrowers can obtain a lower monthly payment in exchange for this upfront payment. Points can also be used to reduce the monthly payment for the purpose of qualifying for a loan based on monthly income. It is important to note that points are different from or...</td>\n",
              "      <td>Points, sometimes also called a 'discount point', are a form of pre-paid interest .</td>\n",
              "      <td>Ragas Context Precision</td>\n",
              "      <td>None</td>\n",
              "      <td>This metric evaluates whether all of the ground-truth relevant items present in the context are ranked higher or not. Ideally all the relevant chunks must appear at the top ranks</td>\n",
              "      <td>1674</td>\n",
              "      <td>gpt-4o</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2e9bc878-68de-4dde-a0c3-575d75b95e7b')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-2e9bc878-68de-4dde-a0c3-575d75b95e7b button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-2e9bc878-68de-4dde-a0c3-575d75b95e7b');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "repr_error": "Out of range float values are not JSON compliant: nan"
            }
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "![corrective.png]()"
      ],
      "metadata": {
        "id": "uN_ljktfK7VG"
      }
    }
  ]
}